Parallel echo version of media content for comment creation and delivery

ABSTRACT

Methods, systems, and devices are described for associating comments with playback of media content. At an input device, a parallel echo version of media content being played on a second device is provided, and a selection is received of a portion of the parallel echo version of the media content. A comment associated with the selected portion of the parallel echo version of the media content is received, and the received comment is associated with a portion of the media content associated with the selected portion of the parallel echo version. At an output device, a parallel echo version of media content being played on a second device is provided, a current point in the parallel echo version of the media content is determined, and an indicator of a comment associated with a point in the media content corresponding to the current point in the parallel echo version is displayed.

CROSS REFERENCES

The present application claims priority to the following patentapplications: U.S. Provisional Patent Application No. 61/421,945, filedDec. 10, 2010, entitled “DYNAMIC MULTIMEDIA COMMENT AND DISPLAYARCHITECTURE”; U.S. Provisional Patent Application No. 61/432,395, filedJan. 13, 2011, entitled “SYSTEM, DEVICE, AND INTERFACE ARCHITECTURE FORCOMMENT CREATION”; U.S. Provisional Patent Application No. 61/432,397,filed Jan. 13, 2011, entitled “SCREEN LOCATION COMMENT ARCHITECTURE”;U.S. Provisional Patent Application No. 61/439,189, filed Feb. 3, 2011,entitled “USER SELECTABLE COMMENT FILTER AND RELATED ARCHITECTURE”; andU.S. Provisional Patent Application No. 61/439,196, filed Feb. 3, 2011,entitled “COMMENT DELIVERY ARCHITECTURE,” each of which is incorporatedby reference in its entirety for all purposes.

The present application is further related to U.S. patent application,filed concurrently herewith, entitled “ASSOCIATING COMMENTS WITHPLAYBACK OF MEDIA CONTENT” (Attorney Docket P003.01 (77828.0009)); U.S.patent application, filed concurrently herewith, entitled “RECOGNITIONLOOKUPS FOR SYNCHRONIZATION OF MEDIA PLAYBACK WITH COMMENT CREATION ANDDELIVERY” (Attorney Docket No. P003.02 (77828.0010)); U.S. patentapplication, filed concurrently herewith, entitled “ASSOCIATION OFCOMMENTS WITH SCREEN LOCATIONS DURING MEDIA CONTENT PLAYBACK” (AttorneyDocket No. P004.01 (77828.0012)); U.S. patent application, filedconcurrently herewith, entitled “MEDIA CONTENT CLIP IDENTIFICATION ANDCOMBINATION ARCHITECTURE” (Attorney Docket No. P006.01 (77828.0013));and U.S. patent application, filed concurrently herewith, entitled“COMMENT DELIVERY AND FILTERING ARCHITECTURE” (Attorney Docket No.P007.01 (77828.0014)); each of which is incorporated herein in itsentirety for all purposes.

BACKGROUND

The present invention relates to the creation, storage, and distributionof commentary on media content.

Most people interact quite regularly with recorded media content, suchas videos, music, books, and the spoken word. Modern society leansheavily on recorded media content to provide both entertainment andeducation. With the recent proliferation of portable media players,smartphones, tablet computers and the like, the demand for recordedmedia content continues to increase.

For many people, social interaction enhances the experience of viewingor listening to recorded media content. For example, the reactions andopinions of one's peers with respect to a certain film may increase thatperson's enjoyment of the film. In other examples, the perspective of afilmmaker or critic with a unique understanding of a particular filmmay, when shared, add greater meaning to one's perception of the film.

Often, a person consuming recorded media content may wish to sharecomments with other consumers of the recorded media content at presentor in the future. Additionally, it may be desirable to incorporatecomments generated by others with regard to specific portions ofrecorded media content into the experience of viewing and/or listeningto the media content.

SUMMARY

Methods, systems, and devices are described for associating commentswith playback of media content.

In a first set of embodiments, a method of associating comments withplayback of media content includes providing at a first device aparallel echo version of media content being played on a second device,receiving a selection of a portion of the parallel echo version of themedia content at the first device, receiving a comment associated withthe selected portion of the parallel echo version of the media contentat the first device, and associating the received comment with a portionof the media content associated with the selected portion of theparallel echo version of the media content.

In a second set of embodiments, an apparatus for associating commentswith playback of media content includes an echo version module, aselection receiving module, a comment receiving module, and anassociation module. The echo version module is configured to provide aparallel echo version of media content being played on a second deviceseparate from the apparatus. The selection receiving module isconfigured to receive a selection of a portion of the parallel echoversion of the media content. The comment receiving module is configuredto receive a comment associated with the selected portion of theparallel echo version of the media content. The association module isconfigured to associate the received comment with a portion of the mediacontent associated with the selected portion of the parallel echoversion of the media content.

In a third set of embodiments, a method of associating comments withplayback of media content includes providing at a first device aparallel echo version of media content being played on a second device,determining a current point in the parallel echo version of the mediacontent provided at the first device, and displaying at the first devicean indicator of a comment associated with a point in the media contentcorresponding to the current point in the parallel echo version of themedia content provided at the first device

BRIEF DESCRIPTION OF THE DRAWINGS

A further understanding of the nature and advantages of the presentinvention may be realized by reference to the following drawings. In theappended figures, similar components or features may have the samereference label. Further, various components of the same type may bedistinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

FIG. 1 is a block diagram of an example system including componentsconfigured according to various embodiments of the invention.

FIG. 2 is a block diagram of an example display with an interface formaking comments according to various embodiments of the invention.

FIG. 3A is a block diagram of an example display with an interface formaking comments according to various embodiments of the invention.

FIG. 3B is a block diagram of an example display with an interface formaking comments according to various embodiments of the invention.

FIG. 3C is a block diagram of an example display with an interface formaking comments according to various embodiments of the invention.

FIG. 4 is a block diagram of an example system including componentsconfigured according to various embodiments of the invention.

FIG. 5 is a block diagram of an example system including componentsconfigured according to various embodiments of the invention.

FIG. 6A is a block diagram of an example display of a standard versionof media content according to various embodiments of the invention.

FIG. 6B is a block diagram of an example display of a parallel echoversion of media content according to various embodiments of theinvention.

FIG. 7 is a block diagram of an example system including componentsconfigured according to various embodiments of the invention.

FIG. 8 is a block diagram of an example input device includingcomponents configured according to various embodiments of the invention.

FIG. 9 is a block diagram of an example output device includingcomponents configured according to various embodiments of the invention.

FIG. 10 is a block diagram of an example system including componentsconfigured according to various embodiments of the invention.

FIG. 11 is a flowchart diagram of an example method of associatingcomments with playback of media content according to various embodimentsof the invention.

FIG. 12 is a flowchart diagram of an example method of associatingcomments with playback of media content according to various embodimentsof the invention.

FIG. 13 is a flowchart diagram of an example method of associatingcomments with playback of media content according to various embodimentsof the invention.

FIG. 14 is a flowchart diagram of an example method of associatingcomments with playback of media content according to various embodimentsof the invention.

FIG. 15 is a schematic diagram that illustrates a representative devicestructure that may be used in various embodiments of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

Methods, systems, and devices are described for associating commentswith playback of media content. In the present description, an inputdevice provides a parallel echo version of media content being played ona second device, and receives a selection of a portion of the parallelecho version. A comment associated with the selected portion of theparallel echo version of the media content is received, and the receivedcomment is associated with a portion of the media content associatedwith the selected portion of the parallel echo version.

An output device provides a parallel echo version of media content beingplayed on a second device, and determines a current point in theparallel echo version of the media content. An indicator of a commentassociated with a point in the media content corresponding to thecurrent point in the parallel echo version is displayed at the outputdevice. In certain examples, the functions of the input device and theoutput device may be performed by a single user device.

This description provides examples, and is not intended to limit thescope, applicability or configuration of the invention. Rather, theensuing description will provide those skilled in the art with anenabling description for implementing embodiments of the invention.Various changes may be made in the function and arrangement of elements.

Thus, various embodiments may omit, substitute, or add variousprocedures or components as appropriate. For instance, it should beappreciated that the methods may be performed in an order different thanthat described, and that various steps may be added, omitted orcombined. Also, aspects and elements described with respect to certainembodiments may be combined in various other embodiments. It should alsobe appreciated that the following systems, methods, devices, andsoftware may individually or collectively be components of a largersystem, wherein other procedures may take precedence over or otherwisemodify their application.

Systems, devices, methods, and software are described for the creationof commentary on multimedia and other information content. In one set ofembodiments, shown in FIG. 1, a system 100 includes input devices 105(e.g., mobile device 105-a, mobile phone 105-b, laptop 105-c, tablet105-d, computer 105-e, or other computing devices), networks 110,central server computer system 115, data store 120, and output devices125 (e.g., mobile device 125-a, mobile phone 125-b, laptop 125-c, tablet125-d, computer 125-e, or other computing devices). Each of thesecomponents may be in communication with each other, directly orindirectly.

Time, screen location, and object-specific comments may be created formultimedia and other information content. A user interface of an inputdevice 105 allows a commentator to generate a comment, associated with atime code, for example, relating to the content (which, for purposes ofthis disclosure, may include movie, TV show, Internet and other video,book, article, song or other audio recording, photograph or other image,commercial advertisement, video game, immersive media, augmented orartificial reality media, the content contained in a comment, or otherdisplayed content). This created comment may be viewable to otherswatching the content in real time, or on a delayed basis. By using atime code, modified time code or other locator or combination oflocators as an external reference, the comment need not actually beinserted into the multimedia, but may be called up from remote serversas the user reaches the applicable point. The comment may include text,video, audio, photographs and other images, graphical overlays,animations, musical notations, geographic coordinates, discussionthreads, external and internal links and associations with media,meta-media or other comments, software applications and applets, specialnotations to set and grant permissions and define system behaviors orany combination thereof.

The comment may be stored locally (at the device 105, set-top box, orother storage device), or may be transmitted to the central servercomputer system 115 for cataloging and storage in data store 120. Thecentral server computer system 115 may be made up of one or more servercomputers, workstations, web servers, or other suitable computingdevices. The central server computer system 115 may be a cable orsatellite headend. The central server computer system 115 may be fullylocated within a single facility or distributed geographically, in whichcase a network may be used to integrate different components.

Data store 120 may be a single database, or may be made up of any numberof separate and distinct databases. The data store 120 may include one,or more, relational databases or components of relational databases(e.g., tables), object databases, or components of object databases,spreadsheets, text files, internal software lists, or any other type ofdata structure suitable for storing data. Thus, it should be appreciatedthat a data store 120 may each be multiple data storages (of the same ordifferent type), or may share a common data storage with other datastores. Although in some embodiments the data store 120 may be distinctfrom a central server computer system 115, in other embodiments it maybe integrated therein to varying degrees. The created commentary may beintegrated into the underlying multimedia or other information content,or may be stand-alone content to be leveraged with technology allowingthe time stamps to sync with the content as it is played.

The user may be alerted to the existence of a comment during playback ofthe content by the appearance of a viewing window or an icon that can betouched or clicked on output device 125, to reveal its contents. Inother examples, the content and commentary may be separated. Users maytailor their experience by selecting certain commentators, or types ofcommentators and/or comments.

The components of the system 100 may be directly connected, or may beconnected via a network 110 which may be any combination of thefollowing: the Internet, an IP network, an intranet, a wide-area network(“WAN”), a local-area network (“LAN”), a virtual private network, thePublic Switched Telephone Network (“PSTN”), or any other type of networksupporting data communication between devices described herein, indifferent embodiments. A network may include both wired and wirelessconnections, including optical links. Many other examples are possibleand apparent to those skilled in the art in light of this disclosure. Inthe discussion herein, a network may or may not be noted specifically.If no specific means of connection is noted, it may be assumed that thelink, communication, or other connection between devices may be via anetwork.

The following description describes various aspects and options for thesystem. In the discussion below, the comments may be referred to ascomments. Comments, as used herein, are to be interpreted very broadly.Comments may be created by users using input devices 105. Comments maybe viewed and controlled by users using output devices 125. An inputdevice 105 may be an output device 125, as well. Comments may be storedand organized in data store 120. Each feature is set forth for purposesof example only, and may be included or excluded in various embodiments.

A variety of interfaces may be used for comment creation. A commentatormay use these interfaces to input a comment into a movie or TV show (orother media) and to have that comment viewable (if the comment is markedpublic) to anyone else viewing the media content. By using a modifiedtime code as an external reference, the comment may not actually beinserted into the media or into any metadata, but may be called up froma remote server as a viewer reaches that point in the media. There maybe an uninterrupted feed by anticipating and pre-loading any upcomingcomments. The user may be alerted to the existence of a comment duringmedia playback by the appearance of a comment icon that can be touchedor clicked to reveal its contents.

Thus, in one embodiment, there may simply be a time marker, screenlocation, and associated text for a given piece of content. This may bestored separately and independently from the underlying content. FIG. 2illustrates an example block diagram 200 of a display 205 with aninterface for making comments. The display 205 may be on, or integratedwith, a mobile device, mobile phone, laptop, tablet, computer,television, head-mounted display, remote control, or any number of othercomputing or viewing devices. The display 205 may be a touch screen orbe controlled by user inputs into a remote, peripheral, or other mobiledevice. In the illustrated display 205, the locate comment icon 210 isoverlaid on the content. However, in other embodiments, the interfacemay be independent or on another screen or window from the content.

Display 205-a illustrates the interface at Time1, with a comment icon210. A user may click on, or otherwise select, the comment icon 210.Display 205-b illustrates the interface at Time2. A comment entry window215 appears in response to the selection of the comment icon to allowthe user to comment via text entry.

A comment may be set to display for a preset, or configurable, period. Auser may set in and out points so that an entire scene can be referencedby the comment. This may allow the user to ‘capture’ the scene or mediasub-unit completely for reference and for playback or sharing. The usermay set the comment to display at the beginning and end point, and to,optionally, display a scene marker during the span of the scene. Thisscene marker may be distinguishable from the standard comment marker. Auser or the system also may set in and out points in combination withthe identification of an entity or object on screen to capture theentire scene in which that object or entity appears.

As noted, comments are not limited to text, but may include text, video,audio, photographs and other images, graphical overlays, animations,musical notations, geographic coordinates, discussion threads, externaland internal links and associations with media, meta-media or othercomments, software applications and applets, special notations to setand grant permissions and define system behaviors or any combinationthereof. Comments may be made by speaking, and the comment may bereplayed during play of the underlying content. Alternatively, there maybe speech to text conversion for making comments and text-to-speechconversion for listening to comments. A user may insert a video ofhimself or herself with speech commentary, or other video may beinserted. Comments may be placed in specific locations on the display205.

Comments from a number of users may be linked, associated, or otherwiseintegrated to create a comment set. A user may insert video, graphical,or audio overlays designed to be used simultaneously with media contentrather than in separate or separable windows so that the media isaugmented or fully joined by the message content. Comments may containsoftware applications that allow or trigger various actions within localor remote devices, software systems, or devices and systems within andoutside the control of the user or the company or entity deliveringthese services. In this way, a comment can control local devices, forexample, to reduce volume in one viewing window or pause or slow themedia, or provide other services to the user or other entities. Commentsmay be used, for example, to control or administer micropayment systemsfor media or premium commentary. Comments may contain a hyperlink topoints outside the comment systems and they also may contain a link toor an association with a comment within the comment system or they maycontain several potential links offering user a choice of links tomedia, media subunits and other comments in the form of a question,quiz, survey, or other device such as a software player that can linkthe comments and play the resulting media and meta-media. Comments andlink-containing comments may be daisy-chained or linked in ahub-and-spoke or other such arrangement to provide unique ways for usersto move through media or through portions of media.

Comments may be inserted in traditionally non-time-based media such aspaintings, photographs, and architectural renderings in such a way as tocreate simulations or animations that allow this media to take on thedimension of time. A commentary sequence might mirror the layering ofpaint on a canvas, the movement of a focal plane in a photograph fromthe farthest to the nearest point, or the construction sequence of abuilding. Such a commentary time flow simulation may then be used toinvite further commentary along this artificial time line. Similarartificial time lines may also be created within time-based media toalter the time flow of the media or to allow the media to containmultiple time line spurs and other forms. For example, the image on asingle video frame may be commented into multiple successive layers thatrepresent an incremental progression in time to show the construction ofthe movie set shown in the video frame.

Comments may be inserted as header comments designed to appear tosubsequent users in a viewing window associated with a point at the verybeginning of the media and to contain general commentary about a pieceof media not tied to a particular point on a timeline. Comments maycontain messaging devices to allow the user, automatically or not, tosend a message directly to another user or entity. Comments may includea device to exclude any user response from the public commentary ormessage stream so that it is viewable only as a private comment ormessage. Comments may have special status and functionality as supercomments that allow individuals and entities, for example, a wikiservice, to capture and distill various comments in a single comment oran ordered collection of comments, which can then be further vetted andadded to. These super comments also may be filterable into collectionsof commentary that match a user's interests and sensibilities. Variouscomment types and combinations may be clearly marked with symbols,colors, or other methods to alert users to their capabilities andlimitations.

In some embodiments, the windows position may be set automatically, ormay be adjusted or otherwise customized at a user's discretion.Additional tabs and controls may be added to allow a user to selectadditional features described elsewhere herein.

FIG. 3A illustrates an example block diagram 300A of a display 305 withan alternative interface for making comments. This display 305 may be anexample of the display 205 described with reference to FIG. 2. Thedisplay 305 may be on, or integrated with, a mobile device, mobilephone, laptop, tablet, computer, television, head-mounted display,remote control, or any number of other computing or viewing devices. Thedisplay 305 may be a touch screen or be controlled by user inputs into aremote, peripheral, or other mobile device. In the illustrated display305, an insert icon 315 and scrubber control 310 are overlaid on thecontent. However, in other embodiments, the interface may be independentor on another screen or window from the content. The scrubber control310 allows a user to control the insertion timing within the content(e.g., within a 10 minute window).

Display 305-a illustrates the interface at Time1, with an insert icon315 and scrubber control 310 overlaid on the content. A user may use thescrubber control 310 to identify the proper place for comment insertion,and then may click on, or otherwise select, the insert icon 315. Display305-b illustrates the interface at Time2. A comment type window 320appears in response to the selection of the insert icon 315 to allow theuser to comment via text, audio, video, or hyperlink, or insert a pause,or adjust the insertion point.

FIG. 3B is a block diagram 325 of a display 330 with an alternativeinterface for making comments. This display 330 may be an example of thedisplay 205 described with reference to FIG. 2. The display 330 may beon, or integrated with, a mobile device, mobile phone, laptop, tablet,computer, television, head-mounted display, remote control, or anynumber of other computing or viewing devices. The display 330 may be atouch screen or be controlled by user inputs into a remote, peripheral,or other mobile device. In the illustrated display 330, an insert icon315-a, scrubber control 310-a, content window 335, comment stream fromother commentators 340, and comment entry window 345 are in differentwindows within the display 330. However, in other embodiments, eachinterface may be overlaid or integrated or on a screen, or some subsetof this functionality may be on a different device.

A user may use the scrubber control 310-a to identify the proper placefor comment insertion, and then may click on, or otherwise select, theinsert icon 315-a. This display configuration allows a user to view theunderlying content via content window 335, and also look at the commentstream 340 (perhaps filtered), and input comments into the comment entrywindow 345. In some examples, comments from the comment stream 340 maybe dragged and dropped into the comment entry window 345, or otherwiseselected for a comment set or other use. There may be threadeddiscussions, or comments from others may be pulled in.

A user may scrub a pane of commentary forward or back in its timelineand similarly scrub a media pane using software controls such as iconsor touch gestures. Users may be able to control whether commentary andmedia panes are coupled or decoupled and may use coupling and decouplingto control multiple panes independently or together. For example, a usermay be able to scrub both the comment stream window 340 and the contentwindow 335 by swiping in either pane when an interface is set to asynchronization state, such as a ‘hard sync’ state.

A user may be able to set or decouple a ‘hard sync’ with touch gesturessuch as swiping both the commentary stream window 340 and the contentwindow 335 simultaneously in the same direction to set a hard sync stateor by swiping both windows 340, 335 simultaneously in oppositedirections to turn off hard sync. Hard sync may be used to browse tomedia content at a desired location and view the relevant commentary,and/or to browse the commentary to view relevant moments in media. Incertain examples, a hard sync control may be resident in one window sothat swiping one window controls both windows while swiping the otherwindow only controls the window being manipulated, allowing a user tofreely manipulate one window out of synchronization without interruptingthe time progression on another window. Such controls also may be usedto manipulate multiple windows, and the set of windows being manipulatedmay or may not include a media window but may contain only commentary,for example. Other windows, such as the comment entry window 345 alsomay be set to control or be controlled by similar actions. Hard synccontrols may be automatically applied within the user environment or maybe set by the user.

These illustrative controls may allow a user to manage multiple windowseach with multiple time referents and then use further manipulation tomove or associate or perform other actions that allow a comment's timeassociations to be changed or to bridge multiple time associations.Doing so may allow a user to move a comment from one window to anotherto change its location in the media or to drop one comment onto anotherto associate the two comments. User actions to control synchronizationand decoupling may include other gestures and manipulations such astapping a handheld device or using swipe and other hand motions that donot contact a touchscreen but are otherwise sensed, or eye motionsdetected by the device and software or combinations of these gestures.

Resynchronization may be controlled by the user via software controlssuch as tapping in a window that represents the desired synchronizationpoint, or resynchronization may occur automatically when a usermanipulates a window so that it reaches the same point in the timelineas the other window. Automatic synchronization also may then beoverridden with continued manipulation.

The system architecture for comment creation may take on a variety offorms. In some examples, DVD, broadcast, or streaming content may bereceived on a computer, set top box, or television. A user interfacecontrol (e.g., the control functions of the configurations illustratedin FIG. 2, 3A, or 3B) may be in the same screen, but distinct from thecontent (e.g., overlaid, or in separate windows). The user interfacecontrol and the content may be on different devices. The user interfacecontrol may use various mechanisms to associate a timestamp with theentry of a comment. In different embodiments, the user interface controlmay be integrated with content to varying degrees. The control windowmay receive inputs via a remote control, mobile device, keyboard, orother peripheral. The user interface control may control underlyingcontent (i.e., pausing underlying content), or not, and it also mayconvey the underlying content to other devices for viewing and otheractions.

FIG. 3C illustrates an example block diagram 350 of a display 355 withan alternative graphical overlay system for making, and viewing,comments from a single interface or set of interfaces. This display 355may be an example of the display 205 described with reference to FIG. 2.The display 355 may be on, or integrated with, a mobile device, mobilephone, laptop, tablet, computer, television, head-mounted display,remote control, or any number of other computing or viewing devices. Thedisplay 355 may be a touch screen or be controlled by user input into aremote, peripheral, or other mobile device.

Display 355-a illustrates the interface at Time1, with an array ofcircles 360 (which may be referred to as category beacons) overlaid onthe content. These beacons 360 may each represent a type of comment or acategory, or a comment source or source category and each beacon maylight up as the video passes a point at which a comment has beeninserted. If a beacon 360 is pressed and held, the comments filtered bycategory may be shown. The beacons 360 may show a steady glow forcomments that capture a media clip or subunit, with increasingbrightness when multiple commentators have chosen the subunit orportions thereof. Regular comments show as brief pulses. By tapping, orclicking on, any of these buttons the category may be displayed. Bypressing and holding, the user may be taken to a comment viewing window(not shown) that has been filtered by the category. The user cannavigate sideways to other comment windows in other categories, andvertically to comment windows filtered by type of comment, such ashumorous, or subject area, such as cinematography. Certain aspect ratios(not shown) may allow some of the components illustrated to be displayedbelow the video field by user preference. Letterbox video may be shiftedto the very top of the display to provide more room below. A window ofcomments may be displayed on one or more multiple lines, or incombination with other modular windows, such as a graphic to showtraffic in the comment stream.

Display 355-b illustrates the interface at Time2. A user may select theoption of having a graphical overlay appear, allowing the user to havecontrols to view comments, write a comment, scrub to comment insertionpoints, and other tasks. The graphical overlay may include a composecomment 365 set of features, and a current comment 370 set of features.

The compose comment 365 bar may be tapped, and the bar may expand to twolines and continue to expand as text is added. Below ‘send,’ on thesecond line is a pair of icons for audio and video recording of thecomment. ‘In’ may be tapped to insert the comment at the currentlocation in the media. It is not necessary to set an out point, but onecan be set by scrubbing to the desired location in the media andpressing the ‘out’ button. The ‘send’ feature provides sending optionssuch as ‘send as a Tweet,’ ‘send to Facebook,’ ‘send as a message,’ aswell as an option to ‘splice’ this comment to another in a commentdocument, using a splice or linking comment to connect them.

The current comment 370 bar shows the most recent comment passed in thetimeline. ‘Current comment’ may be tapped to view the entire comment.‘Current comment’ may be pressed and held to go to a comment viewingwindow. A small vertical line to the left of the current comment showswhen the current media has been captured within a comment clip, andgives a rough indication when more commentators also have chosen thecurrent scene. The triangles at either end of the bar can be tapped tomove forward and back through the comment timeline. ‘Add’ is a thumbsup, adding the comment to the user's set of comments that will be passedalong to others.

In this example, there is also a unified play, pause, and reverse button375. The button 375 may be designed to use a single control for mediaand other content windows. The default mode may be forward mode. A usermay tap once to play, tap again to pause, tap again to play, and so on.From a paused state, a user may press and hold for 1 second (or othertime period) to play in reverse (indicated, for example, by an audiblesound, 1×). A user may tap to pause, tap to play in reverse, tap topause. From a paused state in reverse mode, a user may press and holdfor 1 second (or other time period) to play in forward mode. Tappingpause and play continues to advance media forward. A user may press andhold again for reverse mode. There may also be fast forward and reverse.In either forward or reverse mode, at play or reverse play speed, a usermay press and hold the play button (e.g., holding for 1 second for 2×speed, holding for 2 seconds for 4× speed, and tapping to pause).

There may also be a split scrubber 380 that lets the user scan the mediaand the comment stream simultaneously, and to have control of eachindividually. To insert or view comments, a user slides a pair oftriangle markers along a double timeline and finds a desired location.The top scrubber controls the media (when the media window is on top orto the left) and the bottom scrubber controls the comment stream belowor to the right. The wide space between the lines controls them bothsimultaneously. Scrubber marks may disappear during finger (or cursor)swipe manipulation between the lines.

As the triangles are moved, they may change color or shading. Thisallows the user to be aware of a change in their position on thetimeline when their movement would not otherwise be obvious. If thetriangles are synchronized, the triangles may change color together,remaining the same color. If the triangles are out of synch, they mayeach change to different colors. This may help make the user aware thatthe user is out of synch when the distances on the timeline would nototherwise make it obvious.

With the split scrubber 380, a user may scan media and commentsindependently and quickly return the media to the place where it waspaused. By tapping (or clicking) one or another of the relevant trianglemarkers, the user can choose to have the media resynchronized to thetimeline location of either the media or the comments window. The usermay also resynchronize by initiating play in the appropriate window.Aside from scanning and browsing, the split scrubber 380 may be used tochange the insertion point or in/out points of a comment. The user mayselect a comment in the message window that he or she previously wrote,and after adjusting the media in the other window to the appropriatepoint, click the ‘in’ or ‘out’ icon, or click and drag the comment ontothe media.

The split scrubber 380 also may be of use with other combinations ofviewing windows, such as media and comment document windows, or commentand comment document windows. The split timeline may also be augmentedwith a third or fourth timeline to control multiple windows. The splitscrubber 380 may also display the in and out points of the currentcomment if the comment pertains to a media clip (as opposed to a commentthat has a single insertion point). It also may contain a space, forexample, between the two lines, in which a finger swipe in eitherdirection may advance the media to fine tune a location. It may containpoints at either end that can be clicked or tapped to advance or reversethe media one frame at a time or advance or reverse the comment streamone comment at a time. These points may additionally be controlled bypressing and holding them to advance media in gradual acceleration up to4×, for example. Fine control of this acceleration may be handled by aseries of press/hold maneuvers, such as one press/hold for 1× speed andtwo press/holds for 2×. The split scrubber 380 also may be used toimprove the speed of certain user operations when the system is unableto keep up with both media and commentary streams and when decoupling isdesired.

A fine tune scrubber bar 385 may consist of a line within a window(horizontal or vertical) designed to focus finger swipe motions into aspecific zone or to be manipulated by mouse control. It may be usedwithout split scrubber 380 (e.g., it may be used in configurations inwhich use of a split scrubber 380 could lead to confusion or in whichthe split scrubber is too bulky). In some examples, a fine tune scrubberbar 385 may be used with each window with timeline-related elements,such as media and commentary, served by its own scrubber. Manipulationof one window may control both windows unless the synchronization ofthese windows is decoupled. Windows may be decoupled by pressing andholding one window while manipulating the scrubber of another. Thewindows can be resynchronized by initiating ‘play’ in either window,which synchronizes play to the chosen window. The disappearing scrubmarker described herein may also have the capability of splitting intotop and bottom halves so that this scrubber can be used as a lessobtrusive split scrubber.

A fine tune scrubber bar 385 may be swiped from right to left, forexample, to advance the media or commentary forward in the timeline.Rapid and repeated swiping may trigger the appearance of a markerindicating speed, such as 2× or 4×. It may also trigger the appearanceof a scrub marker that shows progress along a timeline. When thesemarkers appear, it may be a signal to the user to stop swiping and allowthe content to continue at that speed to a desired point, or manipulatethe markers to move a greater distance. A tap may again pause thecontent so that it can be fine-tuned by finger swipe. Pressing andholding a midpoint of the line may also trigger the display of the scrubmarker and allow it to be manipulated to move greater distances in thetimeline and to its beginning or end. This marker may disappear when theuser makes further fine tune swiping motions. Tapping either end of theline may allow the user to advance frame by frame or comment by commentdepending on the window. Pressing and holding one end of the line mayincrease scrubbing speed by successive increments, accelerating thespeed from 2× to 4×, for example. This may work whether or not the scrubmarker is visible, though it may also cause the scrub marker to appearso the user is aware of motion in the timeline. Releasing this may causethe window's content to pause again, or it may continue at that speeduntil the window is tapped to pause it. Further swiping may return theuser to fine adjustments.

This scrubber bar 385 also may show fine movement in the disappearingscrub marker(s) by changing the color of the marker and the markerhalves in the same manner as the split scrubber. These scrub markers mayalso be manipulated in the manner of the split scrubber above todecouple media from other windows and, by pressing and holding one orthe other, resynchronizing to the timeline of the desired window.

There may also be browse scrubber functionality. Browse mode may beactivated by pressing and holding a point in the media or commentwindow. Both may pause as soon as one is touched. Now either window maybe manipulated by horizontal finger swipes to scrub the media and thecomment stream forward and reverse. Rapid swiping in one direction(e.g., three times) may increase the speed of scrubbing and trigger thedisplay of an optional simple scrubber bar and marker that can bemanipulated to move greater distances in the timeline. If the optionalsimple scrubber bar is not manipulated, it may disappear after a shortinterval. If it is manipulated, once a finger is lifted from thescrubber, it may disappear and finger swiping may be automaticallyreactivated. While this swiping tool may default to simultaneous andsynchronized viewing of both windows, the windows may be decoupled bymanipulating one window with a swipe while pressing and holding theother. Once decoupled, the held window no longer needs to be held, andit stays in its position in the timeline.

In browse mode, there may be one screen anchored in a position in thetimeline if the user hopes to return both windows to this originallocation to continue a viewing session. When it is time toresynchronize, the user can press and hold the window that is currentlyat the desired location, and the windows are synched. This mirrors asimilar operation on the split scrubber in which the windows areresynchronized by pressing and holding the triangle marker associatedwith the window that is in the desired timeline position. If a usermistakenly presses the wrong window, the undo command may return thewindows to their previous positions. The interface also may contain acommand that can return the media to its previous viewing position(s) incase many user actions have transpired since the viewing session wasinterrupted. This command may be a second tier option of the undocommand.

Browse mode may also allow the user to manipulate the media and commentscreens vertically. Vertical strokes in the media window may movethrough other media in the queue or its sub-queues or through othercomment projects underway. Pressing and holding points at the top andbottom edges of the window may move through sub-queues and other ofthese categories. This vertical movement may be helpful when the user isworking across media to link scenes, create timeline-independent commentsets, build annotated music playlists, or create other commentary/mediadocuments. It also allows a user to switch gears and enjoy other media.

The comment window may continue to display commentary from the firstmedia until the user presses and holds the media window to synchcommentary to that media and location on the timeline. Vertical strokesin the comment window move the window through the current list ofcomments arranged in timeline order, and each comment is expandable witha tap. Pressing and holding points at the top and bottom edges of thewindow may move through categories and classes of commentary so the usercan view comments on a certain topic or of a certain type, such ashumorous. Vertical movement in the comment window may be used inconjunction with horizontal movement to find a category (verticalmovement to ‘acting’) and then move horizontally to select a type ofcomment, such as humorous, to display humorous comments about theacting. In windows or operations that benefit from diagonal movement,such as forms of a comment document builder, finger swipes can movescreen objects diagonally and may move through matrices diagonally aswell. Matrices and other mapping devices may also be viewed andnavigated via other constructs in a 3D or simulated 3D environment, andthe element of movement through time may add a fourth dimension to thisnavigation.

Browse mode may be further augmented with controls that allow the userto tap the edges of a viewing window to control incremental movements.In the media window, for example, a tap to the right edge (or aspecifically marked spot on the edge) may advance the media one frame.In the message window, a similar tap may advance the comment stream byone comment. Taps on the top and bottom edges may advance the mediawindow through the queue or sub-queues, and move the comment windowthrough the comment categories. Pressing and holding these edges ormarks may move the material in the window to its beginning or end point,such as the end of a media. Finger swiping may be augmented orsubstituted by comparable mousing controls, such as clicking anddragging within a control bar, or with eye movements, such as two rapidglances toward one edge of the screen when a specific viewing window isselected.

In one set of embodiments, shown in FIG. 4, a system 400 includes inputdevice 105-g (e.g., a computer or television), network 110-c, centralserver computer system 115-a, and data store 120-a. Although in thiscase a central server computer system 115-a, the underlying content maybe accessed from a local source (e.g., DVD or other locally storedmultimedia). This system 400 may be the system 100 of FIG. 1. Each ofthese components may be in communication with each other, directly orindirectly.

The central server computer system 115-a may stream, or otherwisetransmit, video data (or other information data) to the input device.This content may be accessed from data store 120-a.

Time, screen location, and object-specific comments may be created forthe content. A user interface control of an input device 105-g allows auser to generate a comment, associated with a time code, for example,relating to the content (which, for purposes of this disclosure, mayinclude a movie, TV show, Internet and other video, book, article, songor other audio recording, photograph or other image, commercialadvertisement, video game, immersive media, augmented or artificialreality media, the content contained in a comment, or other displayedcontent). By using a time code, modified time code, or other locator orcombination of locators as an external reference, the comment need notactually be inserted into the multimedia. The comment may include text,video, audio, photographs and other images, graphical overlays,animations, musical notations, geographic coordinates, discussionthreads, external and internal links and associations with media,meta-media or other comments, software applications and applets, specialnotations to set and grant permissions and define system behaviors orany combination thereof. The user interface control may be generatedlocally, or served from the central server computer system 115-a to theinput device 105-g via network 110-c.

In one example, the content stream and user interface control areindependent and distinct from each other (even when both are on the samedisplay). In other embodiments, the content stream and user interfacecontrol are overlaid or partially integrated. In still otherembodiments, the content stream and user interface control are tightlyintegrated. The following examples further illustrate the options.

FIG. 5 illustrates a system 500 which includes a content source 505(which may be a DVD or other video disc player, a local or remotestorage device, a central server computer system), a set top box 510, amobile device 105-h (e.g., a tablet, smartphone, remote control, cellphone, or laptop), and a display 530 (e.g., television or otherdisplay). The set top box 510 includes a distribution module 515,content processing module 520, and local wireless device interfacemodule 525. Each of these components may be in communication with eachother, directly or indirectly. This system 500 may be the system 100,400 of FIG. 1 or 4.

In one example, the content source 505 streams media content to the settop box 510, which is stored by the content processing module 520. Thecontent processing module 520 may create an echo version of the mediacontent. As used herein, the terms “parallel echo version” or “echoversion” refer to a representation of the media content displayedconcurrently with a playback of a standard version of the media content.In certain examples, the “echo version” may simply be a lower resolutionor lower bandwidth version of the media content. Additionally oralternatively, the echo version may modify or remove certain aspects ofthe media content, such as replacing visual elements in the mediacontent with outlines or removing a certain range of frequencies from anaudio component of the media content.

In parallel (or substantially in parallel), the content processingmodule 520 may provide a regular or high definition version of thecontent to distribution module 515 for delivery to display 530, whilealso delivering the echo version to the local wireless device interfacemodule 525. Local wireless device interface module 525 may deliver theecho version wirelessly to the mobile device 105-h. The mobile device105-h may control the playback of the echo to time the insertion ofcomments from the mobile device 105-h, and this echo timing may bedistinct and independent from the content playback on the display 530(i.e., the echo version and regular/high definition are notsynchronized). In other embodiments, the playback of the echo versionand the regular/high definition version are synchronized (so thatcontrol to stop, pause, fast forward, rewind at the display 530 (for theregular/high definition version) or the mobile device 105-h (for theecho version or the regular/high definition version) controls bothdisplays).

Local wireless device interface module 525 may generate a user interfacecontrol for delivery to the mobile device 105-h. This user interfacecontrol may be transmitted with the echo version of the content tomobile device 105-h. Alternatively, the user interface control may begenerated locally (e.g., as an App stored at the mobile device 105-h,wherein the echo version is integrated with the user interface controlat the mobile device 105-h.

In certain examples, the functionality of the set top box 510 shown inFIG. 5 may be performed by an alternate device, such as the contentsource 505, a special-purpose internet appliance, a server, a networkrouter, a network switch, and/or or a network access point.

In another example, the content source 505 streams different versions(the echo version and the regular/high definition version) of thecontent to the set top box 510, each of which may be stored by thecontent processing module 520. The content processing module 520 mayprovide the regular or high definition version of the content todistribution module 515 for delivery to display 530, while alsodelivering the echo version to the local wireless device interfacemodule 525. Local wireless device interface module 525 may deliver theecho version wirelessly to the mobile device 105-h. The mobile device105-h may control the playback of the echo version to time the insertionof comments, and this echo timing may be distinct and independent fromthe content playback on the display 530. In other embodiments, theplayback of the echo version and the regular/high definition version aresynchronized. A portion or sample of the echo may be attached to orincluded with the comment when a time code or modified time code orother placement techniques alone would be insufficient to assist withsynchronization or in the absence of a connection to the Internet or tothe appropriate local processing module. A portion or sample of the echoalso may be attached or included with the comment when it may be usefulto a subsequent user as a reference or other tool.

There are a number of other ways an echo version may be pulled out ofthe stream for time code needs. Echo version creation may occur at acentral server computer system, on the set top box 510, at the display530, or at the device 105-h. This echo version may also be designed sothat it satisfies the need for 1) time code synchronization, 2) roughvisual cross-checking to help users ensure that comments match up withdesired input location, 3) scrubbing for fine tuning of commentplacement, 4) delivery to comment servers to cross-check or identifymedia or provide synchronization, 5) faster retrieval, viewing, andselecting of media clips, or 6) any combination of these uses.

This echo version may be 1) a simple time code conveyance such as amedia timer linked to the media content as displayed, 2) a lowresolution version of the media content that stays within the bounds ofcopyright restrictions, 3) a version of the media content that isencoded to be inaccessible beyond time code use, 4) a version of themedia content that is altered to be recognizable by the user but not inits standard and enjoyable form (such as music with a significantsection of the frequency range removed, or such as video with only roughobject outlines), or 5) a version of the media that is altered to beusable only by, for example, a data system designed to match audiowaveforms, equivalent forms of video, and equivalent forms of text andother multimedia, or 6) any combination of these uses. An equivalentform of text, for example, might utilize word counts, word and lettercount sequences, or a very difficult to read font or combination offonts.

FIG. 6A and FIG. 6B illustrate displays 605, 610 from a standard versionand an echo version, respectively, of the same media content. Thedisplays 605, 610 may be examples of the displays 205, 305 describedwith reference to FIG. 2 or FIGS. 3A-C. In the illustrative context ofFIG. 5, the standard version of the media content may be displayed onthe display 530, and the echo version of the media content may bedisplayed on the mobile device 105-h. Each of the standard version andthe echo version may include a separate scrubber bar 615 to allow forasynchronous navigation through the standard version 605 and the echoversion 610. The standard version and the echo version of the mediacontent shown in FIG. 6A and FIG. 6B are in a state of synchronizationfor the purposes of illustration.

The echo version of FIG. 6B replaces objects (e.g., buildings) in thestandard version of the media content with outlines. Doing so may savebandwidth and enforce certain content duplication restrictions in placefor the media content. Alternatively, as described above, the echoversion may simply be a low resolution version of the media content.Indicators 620 of existing comments associated with a current point intime in the echo version of the media content are marked with the letterQ.

A viewer may click on a Q or select an area 625 of Qs to view or receivemore information on the comments. As noted, comments are not limited totext, but may include text, audio, video, links, applications, otherthreads, or any combination thereof. Comments of different types mayhave different letters, or different colors, shapes, effects, fonts,and/or other aspects that distinguish the comments. Comments within a 3Denvironment may be overlaid on the image in ways that make use of 3D todisplay them joined with the media so that objects in the echo versionand comment indicators are able to, for example, occlude each other fromview.

As shown in FIG. 6B, the indicators 620 may be displayed at differentscreen locations within the echo version to represent associationsbetween individual comments and screen locations in the media content.The indicators 620 may represent comments retrieved from a centralserver computer system, such as the central server computer system 115described in FIG. 1 or 4. Additionally or alternatively, the indicators620 may represent comments stored locally at a device displaying theecho version of the media content. A user may view one of the existingcomments by selecting an indicator 620 for that comment using, forexample, a finger tap or a cursor.

The echo version may also be displayed with an insert icon 630 forassociating a new comment with the media content. The insert icon 630may be an example of the insert icon 630 described above with referenceto FIG. 3A. Additionally or alternatively, the echo version may includeother comment insertion controls for associating comments with the mediacontent, for example, as described above with reference to FIG. 3B andFIG. 3C.

The echo version may further include a synchronization icon 635 forinitiating a synchronization request in the device displaying the echoversion. Upon initiating the synchronization request, a current point inthe display of the echo version may be adjusted to bring the echoversion into synchronization with the standard version displayed on theother device.

This synchronization may be implemented in a number of ways. Forinstance, in examples where both the standard version and the echoversion are streamed in parallel from a common set top box or a centralserver computer system, the set top box or central server computersystem may adjust the stream of the echo version to synchronize deliveryof the echo version with delivery of the standard version of the mediacontent. In other examples, such as examples where the standard versionis streamed separately from the echo version, sampling and waveform orother pattern recognition may be used to synchronize the delivery of theecho version with the delivery of the standard version. One such exampleis described in more detail with respect to FIG. 7.

FIG. 7 illustrates a system 700 which includes a content source 505(which may be a DVD or other video disc player, a local or remotestorage device, a central server computer system, and/or the like), adisplay 610 (which may be a, television, a monitor, other displaydevice, and/or the like), a mobile device 105-i (which may be a tablet,smartphone, remote control, cell phone, laptop, head-mounted or eyeglassdisplay, other mobile device, and/or the like), a network 110-d, an echoversion server 115-b, an echo version data store 120-b, a patternrecognition server 115-c, and a pattern recognition data store 120-c.Each of these components may be in communication with each other,directly or indirectly. This system 700 may be the system 100, 400 ofFIG. 1 or FIG. 4. The echo version server 115-b and the patternrecognition server 115-c may be examples of the central server computersystem 115 described with reference to FIG. 1 or FIG. 4. The echoversion data store 120-c and the pattern recognition data store 120-cmay be examples of the data store 120 described with reference to FIG. 1or FIG. 4.

In one example, the content source 505 streams or otherwise transmits astandard version of media content to the display 510. The display 510plays the media content. The echo server 115-b streams an echo versionof the media content from the echo data store 120-b to the mobile device105-i over the network 110-d. The echo version of the media content maybe streamed to the mobile device 105-i independently from the streamingof the standard version of the media content to the display 510.

The mobile device 105-i may obtain a sample of the media content playingon the display 510 by capturing audio and/or video from the display(e.g., through a microphone and camera) at a sampling point. The mobiledevice 105-i may transmit the captured audio or video sample over thenetwork 110-d to the pattern recognition server 115-c, which mayidentify the sampling point in the media content playback at the display510 by comparing the sample to waveform or other pattern recognitiondata in the pattern recognition data store 120-c. The patternrecognition server 115-c may also identify the media content. A varietyof audio and/or video recognition techniques known in the art may beused to facilitate content recognition.

The pattern recognition server 115-c may generate a time code associatedwith the sampling point within the content. Alternatively, a nativeapplication at the mobile device 105-i may perform some of thesefunctions. The mobile device 105-i may determine a current point in theplayback of the media content at the display 510 based on an offsetbetween the determined sampling point and a current time. The mobiledevice 105-i may then communicate with the echo version server 115-band/or jump to a different point in a local cache of echo version datato synchronize the display of the echo version of the media content withthe display of the standard version of the media content at the display510. A user may then view and insert comments at the mobile device 105-iin synchronization with the playback of the standard version of themedia content at the display 510. These comments may be stored locallyat the mobile device 105-i and/or at a comment data store (not shown)accessible over the network 110-d.

Referring now to FIG. 8, a block diagram of an example of an inputdevice 105-j is shown. The input device 105-j may interact with a userduring playback of media content to receive and store comments aboutspecific portions of the media content. The input device 105-j may be anexample of the input device 105 described above with respect to FIG. 1,4, or 7. The input device 105-j may provide a user interface whichallows the user at least to generate comments about media content andassociate the comments with discrete points during the playback of themedia content.

The input device 105-j of the present example includes an echo versionmodule 805, a selection receiving module 810, a comment receiving module815, and an association module 820. Each of these components may be incommunication, directly or indirectly. The echo version module 805 maybe configured to display a parallel echo version of the playback ofmedia content occurring on another device. In one example, the inputdevice 105-j may be a tablet computer in the possession of a user who issimultaneously viewing the playback of the media content on a separatedisplay. The echo version module 805 may receive echo version data overa network or from local storage and display the echo version of themedia content to the user, as described above.

The selection receiving module 810 may interact with the user to receivefrom the user a selection of a portion of the echo version of the mediacontent for insertion of a comment. The selected portion of the echoversion may include a time code and/or a location indicator within theecho version of the media content.

The selection of the portion of the echo version may be received usingany of a variety of means. In certain examples, the user may select theportion during the display of the echo version using a scrubber control(e.g., scrubber 615 of FIG. 6B) or a split scrubber control (e.g., splitscrubber 380 of FIG. 3C). Additionally or alternatively, the user mayselect the portion of the echo version during the display of the echoversion of the media content by tapping a button or making a menuselection at a certain time during the display of the echo version ofthe media content to indicate an insertion point. The insertion pointmay be selected in real time during the playback of the media contentand/or retroactively.

In certain examples, the selection receiving module 810 may receivemultiple selections from the user of multiple points during the playbackof the media content for association with a single comment. Forinstance, as described above, the user may select a first point duringthe playback of the media content for insertion of the comment and asecond point during the playback of the media content to indicate anexpiration or termination of the display of the comment.

The comment receiving module 815 may be configured to receive a commentfrom the user. The user may intend that the comment be associated withone or more portions of the media content corresponding to the selectedportions of the echo version. The comment may include text and/or othercontent. In certain examples, the comment may include recorded audio,video, hyperlinks, animated content, and/or other content. As describedpreviously, some comments may even include software functionality, suchas applications or applets that may be invoked when the comments areviewed. For instance, a comment may include executable code such thatwhen the comment is viewed, the code is executed to display aninteractive activity to the viewer.

In additional or alternative examples, the comment(s) received at thecomment receiving module 815 may be associated with one or more existingcomments to allow a viewer of the comments to move through a daisy chainor other linked organization of comments, as described previously. Thus,when a comment is received at the comment receiving module 815, theinput device 105-j may be further configured to receive a selection of asecond comment from the user for association with the first comment.

Once the selected portion of the echo version has been received by theselection receiving module 810 and the comment has been received by thecomment receiving module 815, the association module 820 may beconfigured to associate the received comment with one or more portionsof the media content corresponding to the selected portion of the echoversion module. Where applicable, the association module 820 may alsoassociate the received comment with one or more other comments. Thisassociation may make it possible for a device or process receiving thecomment to identify the most relevant portions of the media content towhich the comment applies. An association with another comment may allowthe explicit or primary time code of a comment to be of secondaryimportance when placed in a user's media time line and may cause it tobe overridden or adjusted in favor of the time code of the othercomment, or it may be associated with multiple points on a time line tocorrespond with multiple comments. Associating one comment with anothermay also be used as a way to correct the placement of a misplacedcomment or suggest a new placement

The comment may be received at the comment receiving module 815 at adifferent point during the playback of the media content on the separatedisplay device than the point with which the user intends to associatehis or her comment. For example, a user watching a video on the separatedisplay device may select point A for insertion of a comment by tappingan insert icon on the input device 105-j during a portion of the echoversion corresponding to point A. The separate display device need notbe synchronized with the echo version shown on the input device 105-jfor the user to select a comment insertion point based on the echoversion.

The association of the received comment with the portion of the mediacontent corresponding to the selected portion of the echo version may beaccomplished in a number of ways. In certain examples, the comment maybe annotated with one or more pointers to the relevant portion(s) of themedia content. Additionally or alternatively, a record of the relevantportion(s) of the media content may be annotated with a pointer to thecomment. In still other examples, a central database may includereferences to comments and comment insertion points during the playbackof the media content, such that a reference to the received comment inthe central database is associated with a reference to the insertionpoint selected for that comment during the playback of the mediacontent.

In certain embodiments, the input device 105-j may be the same deviceused by the user to view the playback of the media content. For example,the input device 105-j may be a laptop or tablet computer whichsimultaneously plays the media content to the user and produces an echoversion of the media content which allows the user to generate commentsfor association with specific points in the media content.

Referring now to FIG. 9, a block diagram of an example of an outputdevice 125-e is shown. The output device 125-e may interact with a userduring playback of media content to display comments to the user aboutspecific portions of the media content. The output device 125-e may bean example of the output device 125 described above with respect to FIG.1, and/or the set top box 510 described above with reference to FIG. 5.In certain examples, the output device 125-e may also be an example ofthe input device 105 described above with reference to FIG. 1, 4, 7, or8. The output device 125-e may provide a user interface which allows theuser at least to receive and view comments about the playback of mediacontent occurring on, for example, a separate device.

The output device 125-e of the present example includes an echo module805-a, a comment receiving module 905, a display module, and a commentindication module 915. Each of these components may be in communication,directly or indirectly.

The echo version module 805-a may be an example of the echo versionmodule 805 described with reference to FIG. 8. The echo version module805-a may be configured to display a parallel echo version of theplayback of media content occurring separately on another device or onthe output device 125-e. In one example, the output device 125-e may bea tablet computer in the possession of a user who is simultaneouslyviewing the playback of the media content on a separate display. Theecho version module 805-a may receive echo version data over a networkor from local storage and feed the echo version to the display module910 for display to the user.

The comment receiving module 905 may be configured to receive commentsfrom a data store of comments associated with the media content. Thedata store of comments may be locally stored and/or accessed over anetwork. Some or all of the comments may be associated with particulartime codes or other selected points during the playback of the mediacontent. The comments may include text and/or other content. In certainexamples, one or more comments may include recorded audio, video,hyperlinks, animated content, and/or other content. As describedpreviously, some comments may even include software functionality, suchas applications or applets that may be invoked when the comments areviewed. For instance, a comment may include executable code such thatwhen the comment is viewed, the code is executed to display aninteractive activity to the viewer.

In additional or alternative examples, the comment(s) received at thecomment receiving module 905 may be associated with one or more existingcomments to allow a viewer of the comments to move through a daisy chainor other linked organization of comments, as described previously. Thus,when a comment is received at the comment receiving module 905, theoutput device 125-e may be further configured to receive a selection ofa second comment from the user for association with the first comment.

The comment indication module 915 may be configured to monitor thedisplay of the echo version of the media content and the flow ofincoming comments. For each incoming comment associated with a point inthe media content, the comment indication module 915 may cause thedisplay module to display an indicator (e.g., indicator 620 of FIG. 6B)for that comment when a current point in the display of the echo versioncorresponds to the point in the media content with which the comment isassociated.

Referring next to FIG. 10, an example of a system 1000 is shown. In thesystem 1000 of FIG. 10, a user 1005 utilizes a user device 1010 to viewan echo version of media content being played back separately at anotherdevice or another portion of the user device 1010. The user device 1010may further allow the user 1005 to view comments associated with acurrent point in the echo version of the media content, generatecomments with respect to selected portions of the echo version of themedia content, and synchronize the echo version of the media contentwith the playback of the media content occurring at a different deviceor a different portion of the user device 1010.

The system 1000 of the present example includes the user device 1010, anetwork 110-e, a pattern recognition server 115-d, a pattern recognitiondata store 120-d, a comment server 115-e, a comment data store 120-e, anecho data server 115-f, and an echo data store. Each of these componentsmay be in communication, directly or indirectly.

The system 800 may be an example of the system 100 described above withrespect to FIG. 1, the system 400 described above with respect to FIG.4, the system 500 described above with respect to FIG. 5, and/or thesystem 700 described above with reference to FIG. 7.

The network 110-e may be an example of the network 110 described abovewith reference to FIG. 1, 4, or 7. The pattern recognition server 115-d,the comment server 115-e, and the media content server 115-f may beexamples of the central server computer system 115 described above withreference to FIG. 1 or 4. The pattern recognition data store 120-d, thecomment data store 120-e, and the media content data store 120-f may beexamples of the data store 120 described above with respect to FIG. 1 orFIG. 4.

The user device 1010 of the present example may perform thefunctionality of both an input device 105 and an output device 125. Assuch, the user device 1010 may be an example of the input device 105described above with reference to FIG. 1, FIG. 4, FIG. 7, or FIG. 8, theset top box 510 described above with reference to FIG. 5, and/or theoutput device 125 described above with reference to FIG. 1 or FIG. 8.

Similar to the input device 105-j of FIG. 7, the user device 1010 of thepresent example includes an echo version module 805-b for generating anecho version of the separate playback of the media content, a selectionreceiving module 810-a for receiving a selection of a portion of theecho version for insertion of a comment, a comment receiving module815-a for receiving a comment from the user 1005, and an associationmodule 820-a for associating the received comment with a portion of themedia content associated with the selected portion of the parallel echoversion of the media content. The echo version module 805-b may generatethe echo version of the media content from echo data received over thenetwork 110-e from the echo data store 120-f via the echo data server115-f. In certain examples, the echo data store 120-f does not store themedia content itself, only the data for generating an echo version ofthe media content. The echo version of the media content generated bythe echo version module 805-b may be output to a display module 910-afor display to the user 1005.

To associate a new comment with the media content, the user 1005 mayindicate a selected portion of the echo version of the media content tothe selection receiving module 810-a. A time code determination module1030 may determine a time code corresponding to a current point in theecho version of the media content or to some other point in the echoversion of the media content selected by the user 1005. The new commententered by the user may be received by the comment receiving module815-a. The association module 820-a may associate a portion of the mediacontent associated with the selected portion of the echo version withthe new comment. A comment data store module 1035 may communicate withthe comment server 115-e over the network 110-e to store the comment andthe association between the comment and the relevant portion of themedia comment in the comment data store 120-e.

In additional or alternative examples, the comment server 115-e mayperform the functionality of associating the portion of the mediacontent associated with the selected portion of the echo version withthe new comment. For example, the comment server 115-e may receive fromthe comment data store module 1035 a selected time code for the echoversion and the new comment. The comment server 115-e may determine thatthe selected time code for the echo version corresponds to a second timecode for the media content. The comment server 115-e may then associatethe comment with the second time code in the comment data store 120-e.

Similar to the output device 125-e of FIG. 9, the user device 1010 ofthe present example includes the comment data store module 1035, whichmay be an example of the comment receiving module 905 of FIG. 9 forreceiving comments associated with a current point in the echo versionof the media content. As with other embodiments, the comments may bereceived over the network 110-e from the comment server 115-e and/orfrom local storage in the user device 1010. In additional similarity tothe output device 125-e of FIG. 9, the user device 1010 may furtherinclude the echo version module 805-b for generating the parallel echoversion of the media content being played on a separate device orseparate portion of the user device 1010, the display module 910-a fordisplaying the echo version of the media content to the user 1005, and acomment indication module 915-a for monitoring incoming comments andcausing the display module 910-a to display an indicator for a number ofreceived comments that are relevant to the current point in the echoversion. The indicator(s) may overlay the echo version displayed at thedisplay module 910-a.

The user device 1010 may further include a synchronization requestreceiving module 1015, a sampling module 1020, a sample identificationmodule 1025, and a time code determination module 1030 for synchronizingthe echo version of the media content displayed at the user device 1010with the playback of the media content at a separate device or separateportion of the user device 1010.

In one example, the user 1005 may request synchronization of the echoversion by tapping or clicking on a button displayed by the user device1010, or through some other indication. The synchronization requestreceiving module 1015 may receive the synchronization request. Asampling module 1020 may obtain a sample of a portion of the playback ofthe media content on the separate device or separate portion of the userdevice 1010. The sample may be obtained by recording audio and/or videofrom the playback of the media content at a sampling point during theplayback of the media content.

The sample identification module 1025 may transmit the sample to thepattern recognition server 115-d. The pattern recognition server 115-dmay search the pattern recognition data store 120-d for an entrymatching the received sample and return a time code associated with thesampling point. This time code may be returned to the sampleidentification module 1025. The sample identification module 1025 mayforward the time code associated with the sampling point to the timecode determination module 1030, which may determine a current point inthe playback of the media content by the separate device or separateportion of the user device 1010. The current point in the playback ofthe media content may be determined by adjusting the time codeassociated with the sampling point by an offset between the samplingpoint and a current time. The time code determination module 1030 maycause the echo version module 805-b to synchronize to the determinedcurrent point in the playback of the media content.

Referring next to FIG. 11, flowchart of an example method 1100 ofassociating comments with playback of media content is shown. The method1100 may be performed, for example, by the input device 105 describedabove with respect to FIG. 1, 4, 6, 7, or 8, the central server computersystem 115 described above with reference to FIG. 1 or 4, the set-topbox 510 described above with respect to FIG. 5, and/or the user device1010 described above with respect to FIG. 10.

At block 1105, a parallel echo version is provided at a first device ofmedia content being played on a second device. At block 1110, aselection of a portion of the parallel echo version of the media contentis received at the first device. At block 1115, a comment associatedwith the selected portion of the parallel echo version of the mediacontent is received at the first device. At block 1120, the receivedcomment is associated with a portion of the media content associatedwith the selected portion of the parallel echo version of the mediacontent.

Referring next to FIG. 12, flowchart of an example method 1200 ofassociating comments with playback of media content is shown. The method1200 may be performed, for example, by the input device 105 describedabove with respect to FIG. 1, 4, 6, 7, or 8, the central server computersystem 115 described above with reference to FIG. 1 or 4, the set-topbox 510 described above with respect to FIG. 5, and/or the user device1010 described above with respect to FIG. 10.

At block 1205, a low resolution version of media content being played ona second device is provided at a first device. The low resolutionversion of the media content may be a parallel echo version of the mediacontent played on the second device. At block 1210, a synchronizationrequest is received from a user of the first device. At block 1215, asample of a portion of the playback of the media content at the seconddevice is obtained. At block 1220, the sample is transmitted to apattern recognition server. At block 1225, a current point in theplayback of the media content on the second device is determined basedon a response from the pattern recognition server to the transmittedsample. At block 1230, a current point in the low-resolution version ofthe content provided at the first device is updated based on thedetermined current point in the playback of the media content on thesecond device. At block 1235, the first device displays the lowresolution version of the media content in synchronization with theplayback of the media content at the second device.

At block 1240, a selection of a location and time in the low resolutionversion of the media content is received at the first device. At block1245, a comment associated with the selected location and time in thelow resolution version of the media content is received at the firstdevice. At block 1250, a time code associated with the selected time isdetermined at the first device. At block 1255, the received comment andselected location are associated with the determined time code. At block1260, the received comment, selected location, and time code aretransmitted to a data store of comments associated with the mediacontent.

Referring next to FIG. 13, flowchart of another example method 1300 ofassociating comments with playback of media content is shown. The method1300 may be performed, for example, by the output device 125 describedabove with respect to FIG. 1, 9, or 10, the central server computersystem 115 described above with reference to FIG. 1 or 4, and/or by theset-top box 510 described above with respect to FIG. 5.

At block 1305, a parallel echo version of media content being played ona second device is provided at a first device. At block 1310, a currentpoint in the parallel echo version of the media content provided at thefirst device is determined. At block 1315, an indicator is displayed atthe first device of a comment associated with a point in the mediacontent corresponding to the current point in the parallel echo versionof the media content provided at the first device.

Referring next to FIG. 14, flowchart of another example method 1400 ofassociating comments with playback of media content is shown. The method1400 may be performed, for example, by the output device 125 describedabove with respect to FIG. 1, 9, or 10, the central server computersystem 115 described above with reference to FIG. 1 or 4, and/or by theset-top box 510 described above with respect to FIG. 5.

At block 1405, a low resolution version of media content being played ona second device is provided at a first device. The low resolutionversion of the media content may be a parallel echo version of the mediacontent played on the second device. At block 1410, a synchronizationrequest is received from a user of the first device. At block 1415, asample of a portion of the playback of the media content at the seconddevice is obtained. At block 1420, the sample is transmitted to apattern recognition server. At block 1425, a current point in theplayback of the media content on the second device is determined basedon a response from the pattern recognition server to the transmittedsample. At block 1430, a current point in the low-resolution version ofthe content provided at the first device is updated based on thedetermined current point in the playback of the media content on thesecond device. At block 1435, the first device displays the lowresolution version of the media content in synchronization with theplayback of the media content at the second device.

At block 1440, the first device receives a comment associated with atime code for the media content and a screen location in the mediacontent over a network. At block 1445, a determination is made that acurrent point in the low resolution version of the media contentprovided at the first device corresponds to the time code. At block1450, an indicator of the comment is displayed at a screen location inthe low resolution version of the media content that corresponds to thescreen location in the media content associated with the content.

A device structure 1500 that may be used for one or more input devices105, the central server computer system 115, for one or more set-topboxes 510, for one or more output devices 125, or for other computingdevices described herein, is illustrated with the schematic diagram ofFIG. 15. This drawing broadly illustrates how individual system elementsof each of the aforementioned devices may be implemented, whether in aseparated or more integrated manner. Thus, any or all of the variouscomponents of one of the aforementioned devices may be combined in asingle unit or separately maintained and can further be distributed inmultiple groupings or physical units or across multiple locations. Theexample structure shown is made up of hardware elements that areelectrically coupled via bus 1505, including processor(s) 1510 (whichmay further comprise a DSP or special-purpose processor), storagedevice(s) 1515, input device(s) 1520, and output device(s) 1525. Thestorage device(s) 1515 may be a machine-readable storage media readerconnected to any machine-readable storage medium, the combinationcomprehensively representing remote, local, fixed, or removable storagedevices or storage media for temporarily or more permanently containingcomputer-readable information. The communications system(s) interface1545 may interface to a wired, wireless, or other type of interfacingconnection that permits data to be exchanged with other devices. Thecommunications system(s) interface 1545 may permit data to be exchangedwith a network.

The structure 1500 may also include additional software elements, shownas being currently located within working memory 1530, including anoperating system 1535 and other code 1540, such as programs orapplications designed to implement methods of the invention. It will beapparent to those skilled in the art that substantial variations may beused in accordance with specific requirements. For example, customizedhardware might also be used, or particular elements might be implementedin hardware, software (including portable software, such as applets), orboth.

It should be noted that the methods, systems and devices discussed aboveare intended merely to be examples. It must be stressed that variousembodiments may omit, substitute, or add various procedures orcomponents as appropriate. For instance, it should be appreciated that,in alternative embodiments, the methods may be performed in an orderdifferent from that described, and that various steps may be added,omitted or combined. Also, features described with respect to certainembodiments may be combined in various other embodiments. Differentaspects and elements of the embodiments may be combined in a similarmanner. Also, it should be emphasized that technology evolves and, thus,many of the elements are exemplary in nature and should not beinterpreted to limit the scope of the invention.

Specific details are given in the description to provide a thoroughunderstanding of the embodiments. However, it will be understood by oneof ordinary skill in the art that the embodiments may be practicedwithout these specific details. For example, well-known circuits,processes, algorithms, structures, and techniques have been shownwithout unnecessary detail in order to avoid obscuring the embodiments.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flow diagram or block diagram. Although each maydescribe the operations as a sequential process, many of the operationscan be performed in parallel or concurrently. In addition, the order ofthe operations may be rearranged. A process may have additional stepsnot included in the figure.

Moreover, as disclosed herein, the term “memory” or “memory unit” mayrepresent one or more devices for storing data, including read-onlymemory (ROM), random access memory (RAM), magnetic RAM, core memory,magnetic disk storage mediums, optical storage mediums, flash memorydevices or other computer-readable mediums for storing information. Theterm “computer-readable medium” includes, but is not limited to,portable or fixed storage devices, optical storage devices, wirelesschannels, a sim card, other smart cards, and various other mediumscapable of storing, containing or carrying instructions or data.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a computer-readable medium such as a storagemedium. Processors may perform the necessary tasks.

Having described several embodiments, it will be recognized by those ofskill in the art that various modifications, alternative constructions,and equivalents may be used without departing from the spirit of theinvention. For example, the above elements may merely be a component ofa larger system, wherein other rules may take precedence over orotherwise modify the application of the invention. Also, a number ofsteps may be undertaken before, during, or after the above elements areconsidered. Accordingly, the above description should not be taken aslimiting the scope of the invention.

1. A method of associating comments with playback of media content, themethod comprising: providing at a first device a parallel echo versionof media content being played on a second device; receiving a selectionof a portion of the parallel echo version of the media content at thefirst device; receiving a comment associated with the selected portionof the parallel echo version of the media content at the first device;and associating the received comment with a portion of the media contentassociated with the selected portion of the parallel echo version of themedia content.
 2. The method of claim 1, further comprising:synchronizing a display of the parallel echo version of the mediacontent at the first device with the playback of the media content onthe second device.
 3. The method of claim 2, further comprising:receiving a synchronization request at the first device; obtaining asample of a portion of the playback of the media content on the seconddevice; and determining a current point in the playback of the mediacontent on the second device based on the sample; wherein thesynchronizing the display of the parallel echo version of the mediacontent at the first device with the playback of the media content onthe second device comprises updating a current point in the display ofthe parallel echo version of the media content at the first device basedon the determined current point in the playback of the media content onthe second device.
 4. The method of claim 3, further comprising:transmitting the sample to a pattern recognition server; and receivingat the first device the current point in the playback of the mediacontent on the second device from the pattern recognition server inresponse to transmitting the sample.
 5. The method of claim 1, furthercomprising: determining a time code associated with the selection of theportion of the parallel echo version of the media content at the firstdevice; wherein the associating the received comment with the portion ofthe media content comprises associating the received comment with thetime code associated with the selection during the playback of the mediacontent.
 6. The method of claim 1, wherein the parallel echo version ofthe media content comprises a low-resolution version of the mediacontent.
 7. The method of claim 1, wherein the parallel echo version ofthe media content comprises a modified version of the media content inwhich at least one visual element in the media content is replaced by anoutline of the at least one visual element.
 8. The method of claim 1,wherein the parallel echo version of the media content comprises amodified version of the media content in which a range of audiofrequencies has been removed from an audio recording of the mediacontent.
 9. An apparatus for associating comments with playback of mediacontent, the apparatus comprising: an echo version module configured toprovide a parallel echo version of media content being played on asecond device separate from the apparatus; a selection receiving moduleconfigured to receive a selection of a portion of the parallel echoversion of the media content; a comment receiving module configured toreceive a comment associated with the selected portion of the parallelecho version of the media content; and an association module configuredto associate the received comment with a portion of the media contentassociated with the selected portion of the parallel echo version of themedia content.
 10. The apparatus of claim 9, further comprising: asampling module configured to obtain a sample of a portion of theplayback of the media content on the second device; and a time codedetermination module configured to determine a first time codeassociated with a current point in the playback of the media content onthe second device based on the sample; wherein the echo version moduleis further configured to synchronize the display of the parallel echoversion of the media content at the first device with the playback ofthe media content on the second device by updating a second time code atthe first device based on the first time code.
 11. The apparatus ofclaim 9, wherein: the association module is further configured toassociate the received comment with a time code corresponding to theselection of the portion of the parallel echo version of the mediacontent at the first device.
 12. A method of associating comments withplayback of media content, the method comprising: providing at a firstdevice a parallel echo version of media content being played on a seconddevice; determining a current point in the parallel echo version of themedia content provided at the first device; and displaying at the firstdevice an indicator of a comment associated with a point in the mediacontent corresponding to the current point in the parallel echo versionof the media content provided at the first device.
 13. The method ofclaim 12, further comprising: synchronizing a display of the parallelecho version of the media content at the first device with the playbackof the media content on the second device.
 14. The method of claim 13,further comprising: receiving a synchronization request at the firstdevice; obtaining a sample of a portion of the playback of the mediacontent on the second device; and determining a current point in theplayback of the media content on the second device based on the sample;wherein the synchronizing the display of the parallel echo version ofthe media content at the first device with the playback of the mediacontent at the second device comprises updating the current point in theparallel echo version of the media content at the first device based onthe determined current point in the playback of the media content on thesecond device.
 15. The method of claim 14, further comprising:transmitting the sample to a pattern recognition server; and receivingat the first device the current point in the playback of the mediacontent on the second device from the pattern recognition server inresponse to transmitting the sample.
 16. The method of claim 12, furthercomprising: receiving at the first device the comment associated withthe current point in the playback of the media content on the seconddevice from a data store over a network.
 17. The method of claim 12,further comprising: determining a screen location in the media contentassociated with the comment; wherein the displaying the indicator of thecomment at the first device comprises displaying the indicator of thecomment at a screen location in the parallel echo version of the mediacontent corresponding to the screen location in the media contentassociated with the comment.
 18. The method of claim 12, wherein theparallel echo version of the media content comprises a low-resolutionversion of the media content.
 19. The method of claim 12, wherein theparallel echo version of the media content comprises a modified versionof the media content in which at least one visual element in the mediacontent is replaced by an outline of the at least one visual element.20. The method of claim 12, wherein the parallel echo version of themedia content comprises a modified version of the media content in whicha range of audio frequencies has been removed from an audio recording ofthe media content.